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SUMMARY 

The IEA’s Progress in International Reading Literacy Study (PIRLS) is a valuable CONTENTS 

resource for educational researchers and policymakers, but not well understood 

by teachers. To improve the impact of PIRLS data on pedagogy in the classroom, SUMMARY al 
researchers and local teachers worked together on Oxford University’s PIRLS for 

Teachers project. This project aimed to provide teachers in England with good INTRODUCTION al 
guidance on interpreting and using knowledge from PIRLS to improve their own 

teaching of reading in primary schools. Outcomes included podcasts, posters and PIRLS FOR TEACHERS Al 
videos supporting best teaching practices. Education systems participating in PIRLS 

are advised to involve teachers in the interpretation of the PIRLS data to ensure GOALS OF THE PIRLS FOR 
educators can make an informed contribution to the national dialogue. As well TEACHERS PROJECT A 
as developing podcasts, posters or similar materials for practitioners, workshops S585 oan 
or practitioner conferences where teachers can interact with researchers foster WORKING WITH 

valuable dialogue on the best use of the data collected by PIRLS. TEACHERS AND PIRLS 


DATA IN ENGLAND 3 


RESULTS: THE PIRLS 


INTRODUCTION FOR TEACHERS PROJECT 


International large-scale assessments OUTCOMES 4 
(ILSAs) have had an increasing influence 
on the discourse surrounding education DISCUSSION 5 


systems around the world. However, the 
results of these studies tend to have less 
impact on pedagogy in the classroom 
than would be expected. For example, 


POLICY IMPLICATIONS AND 
RECOMMENDATIONS ) 


a recent review of 114 published peer- RECOMMENDATIONS 
reviewed articles on the IEA’s Progress FOR EDUCATION 6 
in International Reading Literacy Study 

(PIRLS) indicated that PIRLS is not often REFERENCES 6 
used to inform reading research, which 

could be one reason for its low impact COLOPHON 7 


on pedagogy in the classroom (Lenkeit et 
al., 2015). 


Teachers often lack knowledge about 
ILSAs (Popat et al., 2017). Previous 
studies have indicated teachers struggle 


F IEA POLICY BRIEF 


to make sense of assessment data (Young, 2006) beyond 
general findings. In the case of ILSAs, such general findings are 
normally restricted to aggregated country rankings. Educators’ 
poor assessment literacy often leads to confusion about how 
ILSA data can and should be used to inform their pedagogy. 


In this policy brief, we present findings from the PIRLS for 
Teachers! project, designed to improve teachers’ assessment 
literacy and knowledge of PIRLS. In 2015, teachers from four 
schools in England worked closely with researchers from 
Oxford University to develop strategies for interpreting and 
using the knowledge from PIRLS 2011 to improve teaching of 
reading in primary schools. 


PIRLS FOR TEACHERS 

The PIRLS for Teachers project was designed to bridge the gap 
between researchers and teachers. It aimed to increase both 
teachers’ assessment literacy and their capacity to understand 
and use the data provided by PIRLS. It also aimed to increase 
researchers’ understanding of the challenges teachers face 
when trying to interpret the results presented by PIRLS, and to 
facilitate dialogue between teachers and researchers, with the 
aim of improving teaching practices for reading. 


Aside from attainment scores for reading literacy, PIRLS 
collects a wide range of information through questionnaires 
about students, their learning environments, and motivation 
and interest in reading, which allows researchers to build a 
comprehensive picture of the state of teaching and learning. 
The PIRLS teacher questionnaire provides important contextual 
information about teaching strategies, reading instruction, 
and classroom materials. The information provided by these 
questionnaires offers an opportunity to discuss and share 
results with teachers, and to jointly interpret the results, based 
on educators’ own knowledge of teaching and their particular 
students. The PIRLS for Teachers research team shared all 
available material with teachers and explained how PIRLS 
is carried out, what it measures and how data is collected. 
The research team wanted to hear teachers’ evaluations of 
the material and establish which research questions were of 
interest and relevance to their own teaching practice. 


Like most education systems participating in the PIRLS study, 
there is little dissemination of PIRLS data or engagement with 
teachers in England beyond the league tables published in the 
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1 Oxford University Centre for Educational Assessment was 
awarded an Economic and Social Research Council (ESRC) Impact 
Acceleration Account award to implement the PIRLS for Teachers 
project. As part of the National Centre for PIRLS in England, the 
research team was inspired by similar initiatives in New Zealand, 
Norway, and Germany to engage with teachers (Lenkeit et al., 
2012; Chamberlain, 2012; Skaftun, 2006). The main aim of the 
award was to facilitate knowledge exchange between practitioners 
and researchers. The project was a case study where teachers and 
researchers worked together on the PIRLS data over a period of 18 
months. 


national media (Twist, 2012). Research has shown that the link 
between assessment studies and their impact on classroom 
pedagogy is minimal (Horn et al., 2015). A review of the websites 
of the main teaching unions in England paints a similar picture. 
Clearly, there is a knowledge gap between the information that 
is available from the PIRLS data and what is used to inform 


teachers about the results. 


PIRLS for Teachers was designed as a knowledge exchange 
project, meaning that the two partners, teachers and 
researchers, are seen as equal experts in different areas, 
sharing their experience and expertise to reach a common 
goal. To achieve this, the project focused on how best to 


GOALS OF THE PIRLS FOR TEACHERS 

PROJECT 

1. Review how other education systems disseminate 
PIRLS findings to teachers, 

2. Engage and familiarize teachers with PIRLS, to enable 

them to gain a better understanding of the study, 

. Identify what constitutes useful knowledge for teachers, 

4. Identify ways in which the information provided by 
PIRLS could be used to improve teaching practice, 

5. Conduct in-depth analysis of PIRLS data depending on 
teachers’ interests, 

6. Develop practitioner materials, in collaboration with 
teachers, to support teaching, 

7. Provide visual 
dissemination. 


ice) 


and online resources for wider 


ADOLSYALLNHSO 


communicate and share knowledge between the two groups. 
While teachers are regularly required to communicate with 
other practitioners, students and parents, researchers are 
only used to communicating their findings to colleagues at 
conferences and through academic journals. Researchers are 
not equally skilled at, or trained in, talking to teachers about 
their research and its implications for teaching and learning. In 
fact, some researchers would argue that it is not part of their 
mandate to disseminate their research findings to practitioners, 
or that they do not know how to do so (Reed, 2016). 


Recognizing this, the Oxford University Centre for Educational 
Assessment (OUCEA) was awarded funding for a project to 
collaborate with teachers with the aim of increasing their 
understanding of the PIRLS study and how the results from 
international assessments could be used to improve teaching 
practice in reading. A second goal was to increase researchers’ 
understanding of the challenges teachers face when trying to 
make sense of PIRLS findings, and to better communicate the 
results to schools and teachers. The project was designed to 
provide a forum for teachers and researchers to discuss which 
research questions could be addressed by the PIRLS findings, 
and which would be of particular interest to teachers when 
working on methods to improve teaching practice. 
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WORKING WITH TEACHERS AND PIRLS 
DATA IN ENGLAND 

The PIRLS for Teachers project researchers worked with eight 
teachers from four different schools in England, with the aim 
of making PIRLS and its findings more accessible and useful 
for practitioners. The teachers involved in the project taught 
at schools with differing sociodemographic characteristics: 
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half the teachers taught in small village schools around 
Oxford, where most children come from higher socioeconomic 
backgrounds, while the other half came from larger towns 
near London, where more than half the families have an 
immigrant background and pupils’ parents often have English 
as an additional language. One school was located in an area 
where most of the students came from less socioeconomically 
privileged backgrounds. The teachers recruited were either 
particularly interested in literacy, in charge of literacy 
development in their school, or had worked with literacy and 
reading in other ways. The participants were invited to attend 
two workshops at the University of Oxford, and to meet the 
research team to share their ideas and knowledge regarding 
PIRLS, teaching and reading instruction. 


At the first workshop, the teachers were introduced to PIRLS, 

the design of the study and the reading framework behind it. 

The reading texts used in the test were also discussed, as were 

the items used to measure reading comprehension processes 

such as: 

1. interpreting and integrating ideas and information, 

2. focusing on and retrieving explicitly stated information, 

3. evaluating and critiquing content and textual elements, 
and 

4. making straightforward inferences based on the text. 

The teachers worked through the reading passages, 

accompanying tasks and marking schemes, in order to better 

understand how PIRLS measures reading. They also discussed 

how they interpreted the questions, text, and suggested 

marking scheme. The teachers were particularly interested 

in comparing the text and question formats with their own 

teaching materials, and commented on which texts they would 

expect to motivate certain groups, for example boys versus 

girls. 


The teachers were also provided with the student background 
questionnaires and worked in groups to discuss the different 
areas measured, such as “interest in reading”. The teachers 
could thus see for themselves which questions students would 
be asked and say to what extent they agreed with statements 
assessing interest in reading such as: “I read only if | have to” 
and “I would be happy if someone gave me a book as a present”. 


After discussing the texts, questions, and background 
questionnaires, the teachers worked in two groups to identify 
which areas they were interested in finding out more about 
(see Appendix at www.iea.nl/sites/default/files/publications/ 
Electronic_versions/PolicyBriefl7App.pdf for more detailed 
information about the teacher workshops). Among the 
questions teachers asked were: “How does reading to students 
affect their motivation, confidence and interest in reading?” 
and “How does reading engagement in a specific PIRLS text 
differ between boys and girls, between those students who 
always speak English at home and those with more or fewer 
resources at home?” 
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To address these questions, the PIRLS 2011 achievement, 
student, and classroom data from England were analyzed 
using the IEA’s IDB Analyzer’. At the second workshop, the 
participants critically reflected on whether the findings from 
these analyses coincided with their own experiences in the 
classroom. The group identified possible implications of these 
findings for the teaching of reading in different settings (such 
as schools with relatively affluent students versus those with 
a higher proportion of less affluent students). Although the 
research team initially expected the teachers to be interested in 
developing online materials about PIRLS, the teachers asked for 
posters they could display in their staff rooms and classrooms. 
Teachers argued that visual aids, such as posters, were more 
likely to prompt discussion about the materials and encourage 
application to teaching practice. The research team therefore 
agreed to develop poster materials for practitioners, and, 
following feedback from the teachers themselves and guidance 
from experts, two posters were uploaded to the OUCEA 
website for further dissemination (see http://oucea.education. 
ox.ac.uk/research/recent-research-projects/pirls-for-teachers/ 
pirls-for-teachers-project-outcomes/). 


RESULTS: THE PIRLS FOR TEACHERS 
PROJECT OUTCOMES 

Teacher input was invaluable in developing the project’s 
practitioner materials and ideas. The materials focused on two 
key themes that the teachers identified as important: 


1. Fostering pupils’ motivation through after-reading 
activities, 

2. Supporting pupils’ reading motivation with instructional 
materials. 


2 An introduction to the IDB Analyzer, and other related video 
tutorials explaining the use of the software, are available at http:// 
www.iea.nl/training 


SUMMARY OF THE PIRLS FOR TEACHERS 
PROJECT OUTCOMES 

The PIRLS for Teachers project: 
http://oucea.education.ox.ac.uk/research/recent- 
research-projects/pirls-for-teachers/ 


Links to posters: 
http://oucea.education.ox.ac.uk/wordpress/wp-content/ 
uploads/2017/01/PIRLS-for-Teachers_Poster_After- 
reading-activities.pdf 


http://oucea.education.ox.ac.uk/wordpress/wp-content/ 
uploads/2017/01/PIRLS-for-Teachers_Poster_Reading- 
resources-motivation.pdf 


Links to podcasts: 
http://oucea.education.ox.ac.uk/research/recent- 
research-projects/pirls-for-teachers/pirls-for-teachers- 
project-outcomes/ 


The two posters produced by the project were intended to 
prompt teachers to reflect on the methods they used to teach 
reading. The first poster, “Fostering pupils’ motivation through 
after-reading activities’, summarized analyses of PIRLS 2011 
data from England. The poster communicated that oral after- 
reading activities, such as talking with peers and answering 
questions about what they have read, were positively related 
to students’ interest in and motivation to read, particularly for 
boys and students with few books at home. The conclusion was 
that it was especially important to encourage oral after-reading 
activities to foster interest in reading and motivate students 
who had limited exposure to books at home. 


The second poster was designed around the theme: “Supporting 
pupils’ reading motivation with instructional materials”. It 
summarized analyses from the PIRLS data that revealed 
that the use of a wide range of different reading resources 
and text genres was positively related to students’ interest 
in, engagement with and confidence in reading, as well as 
students’ motivation to read. More specifically, the analyses 
demonstrated that pupils who had teachers who used seven or 
eight different reading resources were more motivated to read 
than pupils whose teachers used a narrower range of reading 
resources. The results differ between boys and girls, and for 
students of different abilities. High-achieving pupils tended to 
like reading more when their teacher used a wide variety of 
reading resources, and girls demonstrated higher confidence 
in reading when teachers used a higher number of resources. 
Further analyses of the PIRLS data found that boys were more 
motivated to read and lower-achieving pupils were more 
engaged with reading when these groups were exposed to a 
variety of reading resources. 


Despite the positive association between a wider range of 
reading resources, confidence in reading and being motivated 
to read, PIRLS has shown that one in three teachers in England 
uses a limited number of reading resources. One important goal 
in primary school is to teach pupils how to understand different 
text forms and genres (see the English programs of study for 
key stages 1 and 2; Department for Education, 2013). The 
PIRLS analyses prompted the poster summarizing advice and 
recommendations for teachers, and suggesting that they use 
a wide variety of reading resources, as this has been found to 
enhance reading motivation for all pupils, independent of their 
reading ability, and is of particular importance for low-achieving 
pupils, fostering their motivation to read, their engagement 
with, and confidence in reading. In particular, the poster 
communicated that using a wide range of reading resources will 
benefit girls’ confidence in reading and boys’ motivation to read. 


The project team also produced informative podcasts for 
teachers. The first podcast explained the PIRLS for Teachers 
project from the teachers’ perspective (Lenkeit & Robinson, 
2016), while the second presented the researchers’ view 
(Lenkeit, 2016). In the teachers’ podcast, a deputy head 
teacher responded to questions like, “Has the PIRLS for 
Teachers project changed your view of PIRLS or what you know 
about it?” A key comment was that she valued working with 
researchers, as teachers in England rarely get time away from 
school or to work on such projects. She focused on the benefit 
of knowledge exchange between teachers from different 
schools and the researchers, but also challenged researchers 
to share future PIRLS analyses with teachers, designed to 
investigate further some of the questions set by the teachers. 
The researchers’ podcast focused on the research impact and 
importance of being able to translate research knowledge 
into findings relevant to potential users, such as teachers, to 
improve education. As participating education systems can 
include questions in the PIRLS study as national options, the 
responses to these questions may be of more interest and 
relevance to teachers in a country. 


DISCUSSION 

When we set up the PIRLS for Teachers project, our expectation 
was that teachers would request online materials to support 
their teaching. However, we found that teachers preferred 
teacher and classroom posters, as daily visual prompts to focus 
teaching practice, perceiving materials they would have to 
search for online as less accessible. Posters are an accessible 
way of presenting the answers to highly specific research 
questions on topics that are directly related to teachers’ 
instructional practice, and provide an opportunity for teachers 
to reflect on their own teaching and discuss their approach 
with colleagues in the same school. 


Through this project, we were able to inspire teacher interest 
in PIRLS and its potential as a resource for informing teaching 
practice. However, we also found that teachers’ interests did 
not always align with the data that is available in PIRLS. Many 
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had ideas for investigating questions relevant to their local 
contexts that were beyond the scope of the PIRLS dataset. 
Participating education systems may reflect on how they could 
make more and better use of the national options in background 
questionnaires to address the need for more context-based 
knowledge for teachers. 


POLICY IMPLICATIONS AND 
RECOMMENDATIONS 

Although studies such as PIRLS are designed to compare 
national systems and do not provide feedback information to 
individual classes or schools, we strongly agree with Rutkowski 
and Rutkowski (2016) that it is important to include teachers 
in the debate on the use of such data. Sebba (2004) indicated 
that there was little information available on how to involve 
practitioners in research, and the use and impact of research on 
practice was therefore minimal. The exclusion of practitioners 
is leaving schools “data rich” but “information poor” (Horn et 
al., 2015). A review of practitioner material on ILSA studies 
(Popat et al., 2017) indicated that very few of the participating 
countries share PIRLS data with teachers. We recommend 
that teachers should be more involved in interpreting PIRLS 
results. Feedback from the workshops and posters indicated 
that teachers recognized that the collaborative aspect of the 
project had triggered considerable dialogue on how to improve 
reading within their own school contexts, and teachers from 
different schools had communicated alternative ideas and 
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examples from their classrooms. The project demonstrates 
how results from PIRLS and other studies may thus have a 
positive impact on classroom pedagogy. As such, the PIRLS 
for Teachers project has contributed towards improving the 
culture of feedback from PIRLS in England, in a way that makes 
the study relevant for teachers. This included gaining teachers’ 
trust by ensuring that they knew that their input to interpreting 
findings was valued and recognizing and acknowledging their 
needs. The project demonstrated that the PIRLS data is a good 
informational resource for teachers if presented in a way that 
makes the relevant data accessible. Given the time and effort 
involved in conducting studies like PIRLS, careful consideration 
should also be given to the small- and large-scale projects that 
attempt to make findings useful for those at the very heart of 
the teaching and learning process. 


Acknowledging the unique expertise of teachers in interpreting 
results, based on their knowledge of their students, classrooms 
and everyday teaching practice, could increase the use of the 
study results among teachers. It could also encourage teachers 
to implement what they believe are relevant ways of enhancing 
students’ learning. 


As most teachers are interested in enabling their students to 
succeed, encouraging teachers to engage with international 
studies such as PIRLS has the potential to benefit pupils in 
the classroom more than if the interpretation and use of ILSA 
studies are left to policymakers and the media alone. If the main 
focus of the PIRLS study is to inform governments around the 
world how to increase students’ reading comprehension and 
understanding, not involving the professionals who interact 
with pupils in the classroom on a daily basis would appear to 
be a lost opportunity to gain valuable expertise. 


RECOMMENDATIONS FOR EDUCATION 
Based on our project and connected research we would 
recommend the following to systems that participate in 
PIRLS: 

1. Include teachers in the interpretation of the PIRLS data 
when the results are released, to ensure educators 
contribute fully to the national dialogue surrounding 
the results in each country. 

2. Work with teachers in education systems that 
participate in PIRLS to develop podcasts, posters, and 
similar materials, and make these easily available to 
practitioners. 

3. Run workshops or practitioner conferences where 
teachers can interact with researchers, fostering 
valuable dialogue on how the data collected by PIRLS 
can be communicated and used to support good 
teaching practice. 
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